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This  paper  compares  the  performance  of  operational  short-range  ocean  forecast  systems 
developed  under  the  Global  Ocean  Data  Assimilation  Experiment  (GODAE)  -  an 
international  effort  to  demonstrate  the  feasibility  of  operational  ocean  forecasting.  ‘Best 
estimates'  from  four  different  operational  forecast  systems  (either  analyses,  hindcasts, 
or  nowcasts)  are  inter-compared  for  the  Tasman  and  Coral  Seas,  off  eastern  Australia. 
Systems  considered  include  those  developed  in  Australia,  France,  the  USA,  and  the  UK. 
Each  system  is  compared  to  observations  of  along-track  sea-level  anomaly,  sea-surface 
temperature,  near-surface  velocity,  and  sub-surface  temperature  and  salinity.  All  have 
their  strengths  and  weaknesses,  and  each  system  out-performs  all  others  in  one  aspect  or 
another.  With  few  exceptions,  all  systems  demonstrate  signal-to-noise  ratios  greaterthan 
one  for  all  variables.  Due  to  the  Australian  focus,  in  addition  to  the  best  estimates  from 
the  operational  systems,  operational  forecasts  and  a  delayed-mode  reanalysis  are  also 
inter-compared  using  the  Australian  system.  The  Australian  system  generally  performs 
the  best  for  sea-level  anomaly;  the  French  system  is  best  for  near-surface  velocities;  the 
USA  system  generally  performs  the  best  for  sea-surface  temperature;  and  the  UK  system 
is  best  for  sub-surface  temperature  and  salinity.  These  findings  provide  useful  indicators 
of  deficiencies  in  each  system  and  clear  metrics  by  which  future  developments  should  be 
assessed.  Based  on  these  results  and  other  practical  considerations  the  adoption  of  multi¬ 
model  consensus  forecasting,  using  all  available  forecasts  from  all  systems,  is  promoted  as 
the  most  robust  approach  for  the  user  community.  Such  developments  are  being  pursued 
under  GODAE  OceanView  -  the  successor  to  GODAE.  The  results  show  the  success  of 
GODAE  in  demonstrating  the  feasibility  of  operational  oceanography. 
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INTRODUCTION 

he  primary  goal  of  the  Global  Ocean  Data 
Assimilation  Experiment1,2  was  to  demonstrate 
the  feasibility  of  operational  ocean  forecast¬ 
ing.  Additional  goals  included  the  development 
and  application  of  state-of-the-art  ocean  models  and  data 
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assimilation  methods  for  short-range  open-ocean  forecasts; 
the  provision  of  global  ocean  analyses  and  reanalyses  for 
developing  and  improving  understanding  of  the  oceans; 
improving  assessments  of  the  predictability  of  ocean  systems; 
and  improving  the  design  and  effectiveness  of  the  global 
ocean  observing  system. 

Several  data-assimilating  ocean  modelling  systems  have 
been  developed  under  GODAE  and  are  used  for  operational 
ocean  forecasting  and  ocean  reanalysis  on  either  regional  or  glo¬ 
bal  scales.3  4  These  include  Bluelink  from  Australia;5,6’7  FOAM 
from  the  UK8,9;  HYCOM  from  the  USA1011;  Mercator  from 
France;12  TOPAZ  from  Norway;13  MOVE  and  COMPASS-K 
from  Japan;14  C-NOOFS  from  Canada  ( www.c-noofs.gc.ca/): 
and  the  ECCO  group  (www.ecco-group.org).  All  these  systems 
are  unique  -  in  most  cases  using  different  model  codes,  differ¬ 
ent  model  resolution,  different  domains,  different  parameteri- 
sations,  different  observation  sources,  different  surface  forcing, 
and  different  data  assimilation  methods. 

Prior  to  the  final  GODAE  Symposium  (www.godae.org/ 
Final-GODAE-Symposium.html).  in  November  2008,  an 
internationally  coordinated  effort  to  undertake  model  inter¬ 
comparisons  of  the  operational  ocean  forecast  systems  was 
undertaken.  Results15  focussed  on  the  North  Atlantic,  tropi¬ 
cal  Atlantic,  and  North  Pacific  basins.  They  concluded  that 
the  ocean  dynamics  of  all  systems  were  consistent;  that  the 
wind-driven  circulation  was  satisfactorily  represented  by  all 
systems  considered;  all  systems  reasonably  represented  the 
thermohaline  circulation.  There  were  differences  in  the  rep¬ 
resentation  of  the  eddy- scale  variability  and  further  analysis 
is  required  to  understand  these  differences. 

These  activities  did  not  include  a  detailed  inter-comparison 
with  the  Australian  system,  because  the  Australian  system 
lacks  sufficient  resolution  in  the  focus  regions  of  that  study. 
It  is  noted  that  the  diversity  of  physical  oceanography  in  the 
Australian  region  that  is  chosen  for  this  study  is  particularly 
interesting  for  assessing  GODAE  systems.  It  includes  a  west¬ 
ern  boundary  current,  with  intense  currents,  fronts,  and  evolv¬ 
ing  (propagating)  features,  strong  horizontal  gradients,  and 
non-linearities.  All  these  features  pose  significant  challenges 
for  models  and  assimilation  schemes,  particularly  where 
observations  only  marginally  resolve  the  features  of  interest. 

The  region  considered  here  also  includes  a  tropical  area, 
the  Coral  Sea,  where  vertical  stratification  and  fast-propagat¬ 
ing  dynamics  are  often  difficult  to  reproduce  with  models. 
Finally,  the  coastal  areas  considered  in  this  study  also  permit 
some  possible  insights  into  how  well  each  system  represents 
the  slope  and  shelf  dynamics,  and  its  interactions  with  open 
ocean  processes. 

The  primary  motivations  for  this  paper  are  two-fold:  firstly, 
to  extend  the  inter-comparisons15  to  the  Australian  region,  and 
to  include  comparisons  with  the  Australian  system;  secondly, 
to  quantify  the  performance  of  GODAE  systems  at  the  end  of 
GODAE  in  2008,  by  comparing  each  system  to  observations. 
Notably,  developments  on  GODAE  systems  continue  under 
GODAE  Ocean  View  (www.godae-oceanview.org).  the  suc¬ 
cessor  to  GODAE. 

The  remainder  of  this  paper  includes  a  brief  description 
of  the  models  and  assimilation  systems,  a  comparison  of  the 
products,  the  results  and  a  discussion  of  the  inter-compari¬ 
sons,  and  finally  the  conclusions. 


SYSTEM  DESCRIPTIONS 

The  important  elements  of  the  models  and  assimilation  systems 
used  in  the  forecast  systems  are  presented  in  Table  1.  Detailed 
descriptions  of  each  assimilation  system  are  available  in  pub¬ 
lished  literature,7’91016  including  summary  papers  that  contrast 
all  assimilation  and  model  systems.3,417  The  descriptions  that 
follow  are  focussed  on  identifying  the  differences  between 
the  systems  that  may  explain  differences  in  the  results,  shown 
later.  These  descriptions  only  include  those  aspects  directly 
relevant  to  the  specific  versions  of  each  system  that  were  used 
for  the  final  GODAE  inter-comparison  exercise.  However,  it  is 
noted  that  development  on  each  system  continues. 

Model  overview 

Of  the  systems  shown  in  Table  1,  Bluelink  (Australia)  and 
HYCOM  (USA)  are  eddy-resolving  in  the  region  of  interest 
(1/10°  and  1/12°  respectively),  and  FOAM  (UKMet)  and 
Mercator  (French)  are  eddy  permitting  (both  1/4°).  Bluelink, 
FOAM  and  Mercator  all  use  z-level  models,  while  HYCOM  uses 
a  hybrid,  adaptive  vertical  grid  (www.hycom.org).  Both  FOAM 
and  Mercator  use  the  same  model  code  and  grid  (NEMO:  www. 
nemo-ocean.eu/).  Bluelink  uses  MOM4.18  All  systems  use  dif¬ 
ferent  in-house  or  national  numerical  weather  prediction  (NWP) 
flux  products.  The  heat  and  freshwater  fluxes  for  Bluelink  and 
FOAM  include  diurnal  variability,  while  Mercator  and  HYCOM 
use  daily  averaged  heat  and  freshwater  fluxes.  Similarly,  the 
wind  forcing  for  Bluelink,  FOAM,  and  HYCOM  resolves  diur¬ 
nal  variability,  but  Mercator  uses  daily  averaged  winds. 

Assimilation  overview 

Each  system  (Table  1)  uses  a  different  method  for  data  assimila¬ 
tion.  Both  Bluelink  and  Mercator  use  ensemble-based  schemes 
to  represent  the  background  error  covariance  -  Bluelink  uses 
Ensemble  Optimal  Interpolation  (EnOI) 19,20  and  Mercator  uses 
a  variant  of  the  Singular  Extended  Evolutive  Kalman  (SEEK) 
filter.21  Bluelink  uses  a  time-invariant  ensemble  (BODAS),7 
and  Mercator  uses  a  seasonally- varying  ensemble  (SAM2).12 
Neither  the  Bluelink  nor  Mercator  ensembles  are  state-depend¬ 
ent  (ie,  not  true  ensemble  Kalman  filters). 

By  contrast,  both  FOAM  and  HYCOM  use  an  optimal 
interpolation  approach  that  uses  some  form  of  analytical 
function  to  approximate  the  background  error  correlations 
derived  from  either  the  differences  between  model  back¬ 
ground  fields  and  observations,  or  from  model  forecast  fields 
of  different  lengths  that  are  valid  at  the  same  time.  The 
FOAM  system  uses  the  Analysis  Correction  (AC)  approach,22 
together  with  the  so-called  National  Meteorological  Center 
(NMC)  approach.23  HYCOM  uses  a  system  called  NCODA, 
a  multivariate  optimal  interpolation  system.10 

The  observations  assimilated  by  each  system  are  quite 
similar  (Table  1);  due  to  continuing  GODAE  efforts  at  data 
assembly  centres,  the  observations  available  in  near-real  time 
(NRT)  for  assimilation  into  different  operational  forecast  sys¬ 
tems  are  increasingly  consistent.  Bluelink  does  not  assimilate 
all  available  observations  -  AMSRE  is  the  only  sea  surface 
temperature  (SST)  product  used  by  Bluelink,  and  Bluelink 
does  not  assimilate  sea-level  anomalies  (SLA)  from  GFO. 
HYCOM  and  FOAM  are  the  only  systems  that  assimilate 
temperature  data  from  surface  drifting  buoys.  Bluelink  and 
HYCOM  are  the  only  systems  that  do  not  use  the  First-Guess 
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Bluelink/  Bluelink/  UKMet /  US  NRL / 

BRAN  OMAPS  FOAM  HYCOM 


Country 

Australia 

Australia 

UK 

USA 

France 

Model  code 

MOM4 

MOM4 

NEMO  3.0 

HYCOM 

NEMO  1.09 

Horizontal  grid 

1/10° 

1/10° 

1/4° 

1/12° 

1/4° 

Vertical  grid 

47  levels 

47  levels 

50  levels 

32  hybrid 

50  levels 

Heat  and 
freshwater  fuxes 

ECMWF  6h 

GASP  3h 

UKMO  6h 

NOGAPS  l-d 

ECMWF  l-d 

Wind  forcing 

ECMWF  6h 

GASP  3h 

UKMO  6h 

NOGAPS  3h 

ECMWF  l-d 

Assimilation  System 

BODAS 

BODAS 

AC/OI 

NCODA 

SAM2 

Background  error 
covariance 

Time  invariant 
Ensemble 

Time  invariant 
Ensemble 

NMC 

Flow-dependent 

Gaussian 

Seasonally-varying 

Ensemble 

Scheme 

EnOI 

(1 20-members) 

EnOI 

(72-members) 

FOAM 

MvOI 

SEEK  (200-220 
modes) 

Localising  length- 
scale 

5°  GC99 

8°  GC99 

n/a 

n/a 

200-500km 

SST  data 

AMSRE 

AMSRE 

AVHRR  + AMSRE 
+  AATSR  -l-  In-situ 

AVHRR  + AMSRE 
+  In-situ 

RTG 

SSH  data 

Jason- 1  +  Envisat 

Jason- 1  +  Envisat 

Jason- 1  +  Envisat 
+  GFO 

Jason- 1  +  Envisat 
+  GFO 

Jason- 1  +  Envisat  + 
GFO 

In-situ 

Argo  +TAO* 

Argo  +  XBT  + 
TAO* 

Argo  +  XBT  + 
buoys  +TAO* 

Argo  +  XBT  + 
buoys  +TAO* 

Argo  +  XBT  +TAO* 

FGAT 

No 

No 

Yes 

No 

Yes 

MSL 

1 5-yr  model 

1 5-yr  model 

Ri05 

20-yr  model 

Rio05 

Assimilation  cycle 

7-d 

3-4-d 

l-d 

l-d 

7-d 

Obs  window 
(SST/SLA/TS) 

3/1  l/7-d 

3/1  l/7-d 

5/5/10-d 

l/3/12-d 

71717-6 

Initialisation 

Nudging  (l-d) 

Nudging  (l-d) 

IAU  (l-d) 

IAU  (6-hr) 

Instantaneous 

update 

Version 

2p2 

ipi 

IpO 

Exp  74.2 

PSY3V2R2 

Table  I :  Summary  of  the  model  and  assimilation  systems  for  each  GODAE  system.  All  of  the  acronyms  are  either  defined 
in  the  text  or  follow:  RTG  refers  to  Real-Time  Global  SST  product  (polar.ncep.noaa.gov/sst/);  TAO*  includes  observations 
from  TAO,  PIRATA  and  RAMA;  GC99  refers  to  Gaspari  and  Cohn  (I999)39;  Rio05  refers  to  the  MSL  described  by  Rio  and 
Hernandez  (2004)40 


at  Appropriate  Time  (FGAT)  method,  and  Bluelink  and 
Mercator  do  not  use  Incremental  Analysis  Updating  (IAU).24 

Briefly,  FGAT  is  often  employed  when  observations  from 
some  time  window,  say  over  five  days,  are  assimilated.  When 
FGAT  is  used  the  model  background  field  is  compared  to  the 
observations  at  the  observation  time.  So  a  time- varying  back¬ 
ground  field  is  used.  When  FGAT  is  not  used,  it  is  common  to 
compare  the  model  background  field  at  the  analysis  time,  to 
observations  over  the  whole  time  window.  Also  IAU  relates 
to  model  initialisation.  When  IAU  is  used,  the  increment  to 
the  model  state  (the  adjustment  to  all  variables  computed  by 
the  assimilation  system)  is  added  to  the  model  field  over  mul¬ 
tiple  time  steps.  For  example,  if  the  increment  is  added  over 
10  time  steps,  one  tenth  of  the  increment  is  added  at  each  time 
step  -  though  the  magnitude  of  the  increment  often  varies 
over  the  initialisation  period.24 

Products  compared 

The  inter-comparison  period  spans  1  February  2008  to 
15  May  2008  -  Austral  Summer  and  Autumn.  The  fields 


provided  by  each  group  are  ‘best-estimates’,  plus  one  fore¬ 
cast  product  from  Bluelink.  The  fields  are  the  class  1  fields, 
where  each  group  has  interpolated  model  fields  onto  a 
common  horizontal  grid,  for  a  predefined  set  of  depths.15 
Specifically,  nowcasts  are  compared  of  the  operational 
systems  from  the  UK  Met  Office,  hereafter  referred  to  as 
UKMet;  hind-casts  from  HYCOM  (version  74.2),  Mercator 
(version  PSY3V2R2),  and  Bluelink  (using  OceanMAPS 
version  lpl,  and  BRAN  version  2p2).  OceanMAPS  is  the 
operational  implementation  of  the  Bluelink  system,6  and 
BRAN,  the  Bluelink  ReANalysis,  is  the  reanalysis  version25. 
Best  estimate  is  used  from  OceanMAPS,  hereafter  referred  to 
as  OMAPS -he,  and  3-4  day  real-time  forecasts  from  version 
lpl  of  OceanMAPS,  hereafter  referred  to  as  OMAPS-fc. 

Because  OceanMAPS  is  initialised  5-days  behind 
real-time,  the  3-4  day  forecasts  reported  here  are  the 
model  forecasts  8-9  days  after  initialisation.  BRAN  and 
OceanMAPS  are  configured  slightly  differently.  BRAN 
assimilates  delayed-mode  quality  controlled  altimetry  and 
Argo  T/S  profiles  but  no  XBT  data,  uses  a  7-day  update 
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cycle,  using  120  ensemble  members,  and  is  forced  with  NRT 
surface  fluxes  from  the  European  Centre  for  Medium-Range 
Weather  Forecasts  (ECMWF).  By  contrast,  the  OMAPS 
products  assimilate  NRT  altimeter  data  and  Argo  TS  profiles 
plus  XBT  data  from  the  Global  Telecommunications  System 
(GTS),  uses  a  3-4  day  update  cycle,  using  72  ensemble  mem¬ 
bers,  and  is  forced  with  NRT  surface  fluxes  from  the  Bureau 
of  Meteorology’s  operational  NWP  system  (GASP). 

In  the  next  section  results  from  each  GODAE  system 
are  compared  to  observations  and  observation-based  prod¬ 
uct  -  specifically,  along-track  Sea  Level  Anomalies  (atSLA) 
from  Jason- 1,  Envisat  and  GFO  satellite  altimeters;  L2P 
AMSRE  SST  (www.remss.com/).  surface  velocities  from 
satellite-tracked  surface  drifting  buoys  (www.aoml.noaa. 
gov/phod/dac).  and  potential  temperature  (T)  and  salinity 
(S)  observations  from  the  Argo  program  (www.argo.net). 
using  delayed-mode  Argo  data  where  it  is  available  (accessed 
from  Coriolis  and  US  GODAE  in  June  2009;  duplicates  are 
removed).  Results  from  GODAE  systems  are  also  compared 
to  observation-based  analysis  products.  These  include  grid- 
ded  SLA  maps,  produced  using  optimal  interpolation  (01)  at 
CSIRO  (www.cmar.csiro.au/remotesensing/  oceancurrents/). 
and  a  gridded  SST  product,  also  produced  using  01  under 
the  Group  for  High  Resolution  SST  (GHRSST)  project 
(www.ghrsst.org)  -  specifically  the  Regional  Australian 
Multi-Sensor  SST  Analysis  (RAMSSA).26  Of  these  data,  all 
are  assimilated  into  each  system,  with  the  exception  of  the 
surface  velocity  data  from  drifting  buoys  that  are  only  assimi¬ 
lated  in  the  HYCOM  system. 

RESULTS  AND  DISCUSSION 

Regions 

Inter-comparisons  of  each  system  are  presented  here  for  two 
regions  in  the  south-west  Pacific  Ocean  -  the  East  Australian 


Current  (EAC)  region,  and  the  Coral  Sea  and  Papua  New 
Guinea  (PNG)  region.  These  regions  are  denoted  in  Fig  1. 
The  EAC  region  is  characterised  by  the  strong  poleward 
currents  of  the  EAC  that  flow  adjacent  to  the  east  Australian 
coast  between  about  20°S  and  33°S.  South  of  about  33°S 
the  EAC  typically  separates  from  the  coast  and  degenerates 
into  a  complex  field  of  mesoscale  eddies.25  The  most  salient 
feature  of  the  Coral  Sea  is  a  quasi- stationary  gyre,  centred 
around  148°E  and  12°S,  called  the  North  Queensland  or  Gulf 
of  Papua  Current27.  The  New  Guinea  Coastal  Current  flows 
west-north- west  along  the  northern  coast  of  PNG,  feeding 
the  West  Pacific  warm  pool28.  Both  the  EAC  and  the  Gulf  of 
Papua  Current  are  fed  from  the  westward  flow  of  the  South 
Equatorial  Current. 

Taylor  diagrams 

In  this  section,  modelled  and  observed  estimates  of  oceanic 
properties  are  compared  using  Taylor  diagrams.29  Taylor  dia¬ 
grams  succinctly  represent  the  unbiased  root-mean-squared 
difference  (ie,  the  RMSD  after  the  mean  has  been  removed) 
and  cross-correlation  between  observed  and  modelled  esti¬ 
mates  of  some  quantity.  Taylor  diagrams  also  show  the 
standard  deviation  of  observed  and  model  estimates.  With 
reference  to  Fig  2a,  showing  comparisons  of  atSLA  in  the 
EAC  region,  the  authors  explain  how  Taylor  diagrams  can  be 
interpreted.  Taylor  diagrams  exploit  a  geometric  relationship 
between  RMSD,  cross-correlation  and  standard  deviation 
through  the  use  of  two  over-lapping  polar  coordinates.  Note 
that  the  mean  of  each  estimate  is  removed  prior  to  generation 
of  the  Taylor  diagram,  so  the  diagram  does  not  provide  any 
explicit  comparison  of  the  mean,  or  bias,  of  each  estimate. 
The  radial  distance  of  each  dot,  for  either  a  set  of  model  esti¬ 
mates  or  observations,  from  the  origin  represents  the  standard 
deviation.  Fig  2a  indicates  that  the  standard  deviation  of  the 
observed  atSLA  is  slightly  greater  than  0.15m.  The  estimate 


Fig  I :  Map  of  the  SW  Pacific  Ocean  showing  an 
estimate  of  the  mean  ocean  circulation  over  the 
top  200m  depth,  two  geographic  regions  (Coral 
Sea  &  PNG  region;  and  EAC  region)  referred  to 
in  this  study,  the  distribution  of  surface  drifting 
buoys  (green),  Argo  T/S  profiles  (magenta),  and 
the  XBT  section  (red) 
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of  atSLA  from  BRAN,  for  example,  is  slightly  less  that 
0.15m.  The  arc  distance  from  the  vertical  axis  corresponds 
to  the  cross-correlation  between  each  model  estimate  and  the 
observed  estimate.  The  vertical  axis  corresponds  to  zero  cor¬ 
relation;  the  horizontal  axis  corresponds  to  a  correlation  of  1, 
with  an  inverse  cosine  scaling  in  between.  The  observed  dot 
is  always  along  the  horizontal  axis  because  it  is  perfectly  cor¬ 
related  with  itself.  For  Fig  2a,  the  cross -correlation  between 
the  observed  atSLA  and  BRAN  is  approximately  0.8.  The 
distance  between  the  dot  representing  the  observed  estimate 
and  each  model  estimate  is  the  RMSD.  So  Fig  2a  indicates 
that  the  RMSD  between  the  observed  atSLA  and  BRAN  is 
slightly  less  than  0.1m. 

Taylor  diagrams  can  be  used  to  get  a  sense  for  the  signal- 
to-noise  ratio  of  each  estimate.  If  the  RMSD  exceeds  the 
standard  deviation,  the  signal-to-noise  is  less  than  1.  In  such 
a  case,  the  model  field  is  not  quantitatively  skilful  because  the 


EAC  region:1 49-1 65E,  38-22S 


EAC  region:149-165E,  38-22S 


signal  is  comparable  to  the  noise  (ie,  error).  If  the  standard 
deviation  of  a  model  is  greater  than  the  standard  deviation  of 
the  observations,  then  it  might  be  concluded  that  the  model 
is  noisier  than  the  observations.  This  might  be  due  to  uncon¬ 
strained  instabilities  in  the  model,  or  because  there  is  more 
energy  in  the  model  than  in  the  observations  -  this  might 
occur  if  a  high  resolution  model  is  compared  to  a  coarse 
resolution  observation,  like  a  Real-Time  Global  (RTG)  SST 
analysis  product,  for  example. 

EAC  Region 

Modelled  and  observed  estimates  of  atSLA,  SST,  and  near¬ 
surface  velocities  in  the  EAC  region  are  compared  in  the 
Taylor  diagrams  presented  in  Fig  2.  Similarly,  modelled 
and  observed  estimates  of  sub- surface  T  and  S  in  the  EAC 
region  are  compared  in  Fig  3,  showing  profiles  of  the  RMSD 
between  observed  and  modelled  estimates.  Also  included  in 


EAC  region:149-165E,  38-22S 


b 


EAC  region:149-165E,  38-22S 


d 


Fig  2:  Taylor  diagrams  for  (a)  atSLA,  (b)  AMSRE  SST  (c)  drifter-derived  zonal  velocities,  and  (d)  drifter-derived  meridional 
velocities  in  the  EAC  region.  The  number  of  observations  used  for  each  panel  is  listed  explicitly  (#  obs  and  #  drifters) 
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EAC  region:  149-165E,  38-22S 


Fig  3:  Profiles  of  the  RMSD  for  T  (left)  and  S  (right)  for  each  GODAE  system,  and  for  climatology,  for  the  EAC  region. 
The  number  of  Argo  profiles  used  for  each  panel  is  listed  explicitly  (#  prof) 


Fig  3  are  profiles  of  the  RMSD  between  observed  fields  and 
climatology  (climatology  from  the  CSIRO  Atlas  for  Regional 
Seas  (CARS)).30 

Using  the  RMSD  as  the  most  basic  metric  for  evalua¬ 
tion,  Figs  2  and  3  indicate  that  the  model  estimates  closest 
to  observations  of  atSLA,  SST,  near-surface  velocity,  and 
sub-surface  T/S  in  the  EAC  come  from  BRAN,  HYCOM, 
Mercator,  and  UKMet  respectively.  Both  Mercator  and 
UKMet  tend  to  under-represent  the  variability  of  near- surface 
velocity,  with  standard  deviations  that  are  measurably  less 
than  the  observations  and  the  HYCOM  and  Bluelink  models. 
This  is  expected,  since  both  Mercator  and  UKMet  models 
are  1/4°,  that  is  appreciably  coarser  than  1/10°  and  1/12° 
resolution  used  by  Bluelink  and  HYCOM  in  this  region.  It  is 
also  interesting  to  note  that  UKMet  tends  to  produce  better 
T/S  profiles  compared  to  Mercator,  while  Mercator  tends  to 
produce  better  SLA.  Noting  that  these  systems  use  the  same 
model  code  and  grid,  it  is  logical  to  conclude  that  these  dif¬ 
ferences  are  attributable  to  the  assimilation  schemes  that  have 
presumably  been  ‘tuned’  differently  for  different  variables  - 
though  the  differences  may  also  be  related  to  differences  in 
the  initialisation  and  surface  forcing. 

Considering  only  the  series  of  Bluelink  results,  the 
estimates  from  the  operational  forecast  (OMAPS-fc)  are 
always  less  skilful  than  the  hindcast  products  (BRAN  and 
OMAPS-hc),  as  might  reasonably  be  expected.  However, 
the  delayed-mode  reanalysis  (BRAN)  is  not  always  more 
skilful  than  the  operational  hindcast  product  (OMAPS-hc). 
This  may  be  due  to  the  length  of  each  update  cycle  -  recall 
that  BRAN  assimilates  observations  once  every  7  days,  while 
OMAPS-hc  assimilates  observations  once  every  3  or  4  days 
(with  twice  weekly  updates).  It  is  expected  that  the  more 


frequent  update  cycle  in  OMAPS-hc  translates  to  an  improve¬ 
ment  in  the  system’s  performance  for  some  variables,  since 
instabilities  in  the  EAC  region  are  known  to  develop  and 
evolve  rapidly  over  very  short  periods,  like  2-3  days.  BRAN 
and  OMAPS-hc  also  use  different  surface  fluxes  from  opera¬ 
tional  NWP  systems  (see  Table  1). 

Considering  the  sub-surface  profiles  of  the  RMSD  for 
T  and  S  (Fig  3),  the  results  show  that,  with  the  exception  of 
the  forecast  product,  all  estimates  have  smaller  errors  than 
climatology  over  the  upper  ocean  (<800m  depth).  This  is 
encouraging,  particularly  in  this  region  that  is  characterised 
by  transient,  mesoscale  features.  In  such  a  region,  the 
displacement  of  an  eddy  can  result  in  very  large  RMSD 
statistics. 7Close  inspection  of  Fig  3  near  the  surface  seems 
to  give  a  result  that  is  somewhat  inconsistent  with  the  SST 
comparisons,  presented  in  Fig  2a.  Specifically,  the  RMSD 
of  the  near- surface  T  is  smallest  for  UKMet  in  Fig  3  a,  but 
the  RMSD  of  SST  is  smallest  for  HYCOM  in  Fig  2b.  This 
is  due  to  different  spatial  and  temporal  sampling  of  Argo 
(approximately  200  profiles  in  total  for  the  whole  region  over 
the  entire  time  period)  and  satellite  SST  (approximately  25km 
resolution  maps  spanning  most  of  the  region  every  day). 
This  can  also  be  attributable  to  the  gradient  in  temperature  in 
the  near-surface  layer  and  the  different  observing  depths  of 
in- situ  (~lm)  and  satellite  (microns)  measurements. 

Coral  Sea  and  PNG  region 

Modelled  and  observed  estimates  of  atSLA,  SST,  and 
near-surface  velocities  in  the  Coral  Sea  and  PNG  region 
are  compared  in  the  Taylor  diagrams  presented  in  Fig  4. 
Similarly,  modelled  and  observed  estimates  of  sub-surface 
T  and  S  in  the  Coral  Sea  and  PNG  region  are  compared  in 
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Coral  Sea  and  PNG  region:142.5-166E,  22-OS 


Coral  Sea  and  PNG  region:1 42.5-1 65E,  22-OS 


Fig  4:  As  for  Fig  2,  except  for  the  Coral  Sea  and  PNG  region 

Fig  5,  showing  profiles  of  the  RMSD  between  observed 
and  modelled  estimates. 

The  RMSD  statistics  shown  in  Figs  4  and  5  again  indicate 
that  the  model  estimates  closest  to  observations  of  atSLA, 
SST,  near-surface  velocity,  and  sub-surface  T/S  in  the 
Coral  Sea  and  PNG  region  come  from  BRAN,  OMAPS-hc, 
Mercator,  and  UKMet  respectively.  For  the  meridional  veloc¬ 
ities,  both  Mercator  and  UKMet  again  tend  to  under-represent 
the  variability  (Fig  4d),  with  smaller  standard  deviations  than 
the  other  estimates. 

In  this  region,  the  RMSD  between  observed  and  modelled 
sub- surface  T  for  all  systems  are  significantly  less  than  cli¬ 
matology  for  depths  shallower  that  400m  -  the  depths  that 
are  of  particular  interest  for  GODAE  systems  (and  indicative 
of  the  depths  constrained  by  the  remotely  sensed  observing 
system).  However,  the  RMSD  for  S  is  not  very  much  less 
than  climatology.  This  suggests  that  S  is  not  particularly  well 
constrained  by  any  of  the  models  in  this  region  -  perhaps 


Coral  Sea  and  PNG  region  :1 42.5-1 65E,  22-OS 


b 


Coral  Sea  and  PNG  region:1 42.5-1 65E,  22-OS 


d 


because  there  are  too  few  observations  and  also  they  are  less 
correlated  with  the  remotely  sensed  observations.  It  could 
also  be  an  indication  that  the  assimilation  systems  are  gener¬ 
ally  not  well  tuned  for  S. 

XBT  transect 

The  frequently  repeated  XBT  transect  PX-6,  between 
Auckland  and  Fiji  (Fig  1),  was  occupied  during  the  inter¬ 
comparison  period.  The  observed  T  section  from  this  line  is 
shown  in  Fig  6a.  Comparisons  of  T  anomalies  (ie,  anomalies 
from  CARS)  at  several  depths,  based  on  an  agreed  subset  of 
depths,31  are  also  shown  in  Fig  6b-e.  The  RMSD  between 
the  observed  and  modelled  T  for  these  comparisons  is 
shown  in  Table  2.  These  comparisons  demonstrate  that  the 
GODAE  models  generally  provide  a  realistic  representation 
of  the  sub-surface  T  field.  Anomalies  along  this  section  range 
from  -1.2°C  to  2.1°C;  and  the  GODAE  systems  generally 
represent  the  sign  and  locations  of  the  anomalies,  and  the 
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Coral  Sea  and  PNG  region:  1 42.5-1 65E,  22-OS 


Fig  5:  As  for  Fig  3,  except  for  the  Coral  Sea  and  PNG  region 


approximate  magnitude  of  the  anomalies  along  the  transect. 
Consistent  with  the  comparisons  in  Figs  3  and  5,  the  results 
from  UKMet,  along  with  OMAPS-hc,  appear  to  out-perform 
the  others. 

SLA  sequence 

A  sequence  of  SLA  maps  are  shown  in  Fig  7,  including  a 
model-independent  analysis  product  as  a  reference  that  is  based 
on  optimal  interpolation  (labelled  OI-SLA),  and  SLA  fields 
from  each  system.  The  purpose  of  this  comparison  is  simply  to 
demonstrate  that  all  systems  produce  realistic  estimates  of  the 
time-varying  state  of  the  ocean.  During  the  period  displayed 
in  Fig  7,  the  salient  features  include  two  warm-core  eddies 
and  three  cold-core  eddies,  all  in  the  vicinity  of  31-37°S.  All 
systems  include  some  representation  of  each  of  these  features  - 
though  their  evolution  over  time  differs.  Specifically,  HYCOM 
shows  the  two  warm-core  features  being  drawn  together,  with  a 
hint  that  they  will  coalesce.  The  variability  in  the  UKMet  fields 
is  notably  less  than  the  OI-SLA  and  the  models.  Mercator, 
BRAN,  and  OMAPS-hc  all  show  the  south-eastern-most  cold- 
core  eddy  splitting  into  two  by  26  February  -  a  feature  that  is 
also  evident  in  the  OI-SLA,  but  not  in  HYCOM  or  UKmet. 
The  differences  between  the  models  occur  in  the  positions 
and  intensity  of  mesoscale  features.  Notably,  the  perform¬ 
ance  of  OMAPS-fc,  the  only  true  operational  forecast  product 
considered  in  this  study,  shows  good  agreement  with  the  other 
products,  including  the  OI-SLA. 

SST  sequence 

A  sequence  of  SST  maps  (Fig  8)  compares  a  GHRSST 
analysis  product  (RAMSSA),26  with  daily  mean  SST  fields 


Depth 

Im 

50m 

200m 

400m 

HYCOM 

0.36 

0.65 

0.56 

0.74 

UKMet 

0.27 

0.52 

0.60 

0.74 

BRAN 

0.45 

0.64 

0.73 

0.68 

Mercator 

0.32 

0.65 

0.58 

0.61 

OMAPS-fc 

0.50 

0.65 

0.90 

0.83 

OMAPS-hc 

0.39 

0.50 

0.60 

0.61 

Table  2:  RMSD  between  T  in  each  GODAE  system  and  T 
along  the  PX-6  XBT  line  (with  the  mean  difference  removed 
in  each  case;  see  Fig  6)  at  various  depths 


from  HYCOM,  UKMet,  Mercator,  BRAN,  and  OMAPS.  In 
general,  all  of  the  GODAE  systems  reproduce  SST  fields 
that  compare  well  with  the  GHRSST  analysis.  The  SST 
fields  from  UKMet  and  Mercator  are  slightly  smoother  than 
GHRSST,  HYCOM,  and  the  Bluelink  products  -  though  one 
could  argue  that  they  are  less  noisy  and  therefore  possibly 
more  reliable.  There  are  times  when  several  systems  appear 
to  have  somewhat  noisy  SST  fields.  For  example,  note  the 
small-scale  filaments  around  160°E  and  35°S  on  9  April 
in  UKMet,  Mercator,  and  BRAN.  Small-scale  features  are 
also  evident  in  GHRSST  and  HYCOM  -  though  the  corre¬ 
spondence  between  those  two  products  is  very  good.  More 
generally,  all  model  products  appear  to  show  somewhat  noisy 
SST  fields  around  35°S. 
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Fig  6:  Comparison  of  T  along  the  PX-6  XBT  transect,  denoted  in  Fig  I ,  that  was  occupied  between  21-25  February  2008, 
showing  (a)  the  latitude-depth  section  from  observations,  and  (b-e)  T  anomalies  (relative  to  climatology)  from  the  XBT 
observations  (bold  black)  and  from  each  GODAE  system  (see  legend)  for  1 ,  50,  200,  and  800m  depth 


At  different  times,  some  GODAE  systems  include  features 
that  appear  fictitious  -  possibly  due  to  assimilation  of  bad 
data,  or  due  to  dynamical  instabilities  in  the  models  that 
are  known  to  occur  in  this  region32.  Again,  it  is  noted  that 
Bluelink  and  HYCOM  offer  higher  resolution  than  UKMet 
and  Mercator.  This  might  explain  some  of  the  differences. 
Moreover,  if  the  GHRSST  product  is  too  smooth,  some  SST 
features  that  appear  in  the  high-resolution  models  might  be 
absent  in  the  GHRSST  product. 

Variability 

The  standard  deviation  of  SLA  from  OI-SLA  and  from 
each  GODAE  system  is  shown  for  the  EAC  region  in  Fig  9. 
The  observation-based  product  (OI-SLA)  shows  six  local 
maxima  in  SLA  variability,  with  a  magnitude  that  exceeds 


0.2m.  OI-SLA  shows  one  local  maxima  around  37°S,  one 
around  36°S  (about  159°E),  two  around  34°S,  and  two 
around  32°S.  All  GODAE  systems  reproduce  the  local 
maxima  around  37°S,  all  reproduce  two  local  maxima 
around  34°S,  and  all  reproduce  two  local  maxima  around 
32°S.  This  result  is  very  encouraging,  because  it  demon¬ 
strates  that  the  variability  reproduced  by  each  system  is 
qualitatively  realistic.  Only  HYCOM  and  BRAN  repro¬ 
duce  the  local  maxima  at  36°S;  and  only  Mercator  and 
BRAN  realistically  reproduce  the  magnitude  of  the  largest 
maximum  (at  34°S,  155°E).  It  is  also  noted  that  the  repro¬ 
duction  of  many  of  these  features  by  OMAPS-fc  is  good. 
This  demonstrates  that  the  GODAE  systems  can  realisti¬ 
cally  forecast  (both  persist,  or  maintain  after  initialisation, 
and  dynamically  evolve)  this  variability. 
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Fig  7:  Sequence  of  daily  averaged  SLA  during  February  2008  from  optimally  interpolated  observations  (OI-SLA;  col  I),  and  each 
GODAE  system  (cols  2-7) 


The  standard  deviation  of  SST  from  the  GHRSST  product 
and  from  each  GODAE  system  is  shown  for  the  south-west 
Pacific  in  Fig  10.  In  general  the  GHRSST  product  has  lower 
variability  than  all  of  the  GODAE  systems.  The  most  promi¬ 
nent  feature  in  the  observation-based  product  (GHRSST)  is 
the  diagonally  oriented  maximum  centred  on  about  155°E 
and  35 °S.  All  of  the  GODAE  systems  reproduce  a  similar 
maximum.  Another  prominent  feature  is  the  band  of  high 
variability  around  22° S  that  extends  along  the  north-east 
coastal  of  Australia,  to  about  15°S. 

All  GODAE  systems  reproduce  relatively  high  variability 
in  these  regions,  though  the  models  (with  the  exception  of 
OMAPS-hc  and  OMAPS-fc)  tend  to  have  greater  variability 
there,  particularly  near  the  coast,  over  the  Great  Barrier  Reef 
(GBR).  BRAN  also  significantly  over-estimates  the  SST  vari¬ 
ability  in  Bass  Strait,  around  40°S.  The  over-estimate  of  SST 


variability  in  these  coastal  regions  (Bass  Strait  and  GBR)  is 
an  indication  that  the  models  are  less  constrained  and  possibly 
lack  some  important  physical  processes  (eg,  tidal  mixing  - 
none  of  the  systems  considered  include  tides).  Also,  the  mod¬ 
els  may  be  overly  sensitive  to  the  quality  of  the  atmospheric 
fluxes.  BRAN,  for  example,  only  assimilates  AMSR-E  SST 
that  is  not  available  within  about  50km  of  the  coast,  and  has 
no  data  in  Bass  Strait  (at  least  none  that  are  assimilated  into 
BRAN).  The  modelled  SST  can  show  higher  variability  and 
frequency  near  the  coast  due  to  strong  local  wind  forcing,  and 
in  the  absence  of  sufficient  observations,  the  modelled  vari¬ 
ability  can  become  somewhat  unreliable.  In  those  locations, 
the  accuracy  of  the  model  is  closely  related  to  the  accuracy  of 
the  local  wind  forcing.  This  was  shown  to  be  true  for  BRAN.7 

In  general,  the  SST  standard  deviation  in  the  models  tend 
to  be  on  small  scales  and  quite  noisy  in  the  southern  part 
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Fig  8:  Sequence  of  daily  averaged  SST  during  March  and  April  2008  from  a  GHRSST  analysis  (col  I)  (RAMSSA31),  and  each 
GODAE  system  (cols  2-7) 


of  the  domain  shown  in  Fig  10.  This  is  an  indication  of  the 
dynamic  nature  of  the  variability  there,  and  may  imply  that 
predictability  in  that  region  is  limited  by  dynamic  instabili¬ 
ties.32  Another  region  of  interest  where  there  is  notable  disa¬ 
greement  with  observations  is  on  the  coastal  side  of  the  Gulf 
of  Papua  Current  (around  10°S).  All  models  show  relatively 
high  SST  variability  there  (Fig  10),  but  the  observations  do 
not.  The  reason  for  this  is  unclear,  but  again,  it  may  be  an 
indication  that  the  models  are  insufficiently  constrained  there 
due  to  a  paucity  of  observations. 

CONCLUSIONS 

The  purpose  of  this  paper  is  to  document  the  performance 
of  GODAE  systems  at  the  end  of  the  GODAE  project,  in 
2008,  and  to  extend  the  inter-comparisons15  to  the  Australian 


region,  and  to  include  inter-comparisons  with  the  Australian 
system.  Comparisons  between  observations  and  GODAE  sys¬ 
tems  demonstrate  that  each  system  has  certain  strengths  and 
weaknesses.  Inter-comparisons  are  regarded  as  valuable  exer¬ 
cises  that  provide  important  insights  into  the  ocean  forecast 
systems  for  both  developers  and  users.  This  inter-comparison 
study,  and  that  of15,  is  one  of  the  first  attempts  at  performing 
GODAE  inter-comparisons.  Now  that  the  tools  have  been 
developed,  the  metrics  have  been  defined,  and  the  community 
has  engaged  in  inter-comparisons,  the  GODAE  Ocean  View 
community  is  well  placed  to  undertake  more  comprehensive 
inter-comparisons,  using  more  complete  datasets;  and  more 
compatible  datasets;  including  class  2,  3  and  4  metrics  to 
assess  predictive  skill;  and  on  the  capacity  for  GODAE  sys¬ 
tems  to  deliver  forecasts  that  are  reliably  skilful  delivering 
real  benefit  the  user  community. 
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Fig  9:  Standard  deviation  of  SLA  from  optimally  interpolated  at  SLA  (OI-SLA)  and  each  GODAE  product 
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Fig  10:  Standard  deviation  of  SST  from  a  GHRSST  analysis  and  each  GODAE  product 


22 


Journal  of  Operational  Oceanography 


Volume  5  No  2  August  20 1 2 


GODAE  inter-comparisons  in  the  Tasman  and  Coral  Seas 


It  is  concluded  that  off  the  east  coast  of  Australia,  the 
Bluelink  (Australian)  system  produces  the  most  accurate  rep¬ 
resentation  of  SLA,  the  HYCOM  (USA)  system  produces  the 
most  accurate  representation  of  SST,  the  Mercator  (French) 
system  produces  the  most  accurate  representation  of  near¬ 
surface  velocity,  and  the  FOAM  (UKMet)  system  produces 
the  most  accurate  representation  of  sub-surface  T  and  S. 

Different  variables  are  important  for  different  applica¬ 
tions.  So,  based  on  this  study  a  user  might  benefit  from  using 
analyses  from  the  system  with  strengths  in  the  variable  most 
relevant  to  their  application.  For  example,  during  a  search 
and  rescue  operation,  a  user  might  adopt  Mercator;  to  identify 
the  position  of  an  eddy,  say  for  a  fisheries  application,  a  user 
might  adopt  Bluelink  (BRAN);  to  monitor  the  chance  of  a 
warming  event  that  might  impact  corals  or  aquaculture,  a  user 
might  adopt  HYCOM  or  Bluelink  (OMAPS);  and  to  predict 
underwater  acoustics  that  depend  strongly  on  sub-surface  T, 
a  user  might  adopt  UKMet. 

However,  noting  that  the  system  with  lowest  error  of  the 
analysis  or  forecast  for  individual  events  varies  with  time 
(ie,  the  systems  contain  independent  information),  a  more 
robust  approach  is  to  combine  information  from  all  sys¬ 
tems  together,  in  the  form  of  a  so-called  super  ensemble.33 
or  a  consensus  forecast.  Given  their  different  strengths  and 
weaknesses,  such  an  ensemble  could  prove  most  useful  for 
communities  that  depend  on  operational  oceanography.  This 
approach  becomes  more  practical  when  considering  the  non¬ 
stationary  nature  of  operational  ocean  forecasting  with  regu¬ 
lar  system  update  cycles  (noted  below)  as  well  as  changes 
to  the  observing  system  and  atmospheric  forcing.  It  is  also 
noted  that  user-oriented  inter-comparison  studies34,35  should 
also  be  performed  to  comprehensively  assess  the  suitability 
of  GODAE  systems  for  different  applications. 

Development  of  the  GODAE  systems  used  here  continues 
under  GODAE  OceanView,  the  successor  to  GODAE,  and  the 
performance  of  each  system  continues  to  improve25,36,37  38  (see 
also  the  GODAE  OceanView  national  reports  at  www.godae. 
or g/documents . htm  and  www. godae-oceanview.org/docu- 
ments/).  Based  on  the  results  of  this  study,  however,  it  is  clear 
that  operational  oceanography  is  indeed  feasible,  and  is  now  in 
full  swing  at  several  agencies.  The  transition  of  these  systems 
into  operational  services  is  being  aided  through  the  WMO/ 
IOC  Joint  Technical  Commission  for  Oceanography  and 
Marine  Meteorology  -  JCOMM  (www.jcomm.info).  As  the 
skill  of  operational  systems  improve  through  adoption  of  new 
observing  systems,  improvements  to  assimilation  methods, 
and  more  accurate  models,  the  impact  of  GODAE  systems 
on  industry  groups  and  marine  users  is  likely  to  continue  to 
increase;  and  through  the  continued  cooperation  of  interna¬ 
tional  partners  under  GODAE  OceanView,  the  potential  to 
produce  accurate,  reliable  ocean  forecasts  is  being  realised. 
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